Repository navigation
ISS-14158: Define when to capture and guard screenshot runs - #435
Merged
Merged
Conversation
Adds the capture guard (refuses repeated main baselines and captures over 240 shots or about 20 minutes, with a fixed-shape ruling override), the fleet.sh --capture slot, the when-to-capture block for visual QA, one-pass visual QA after fixes, and queue-based ETAs. vibe 1.1.91.
Adds two fleet.test.mjs cases: a missing or unreadable capture spec is a usage error before any load check, slot or command, and the capture and containerized e2e build flags are accepted in either order.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ISS-14158: vibe and fleet-orchestrator now say when to capture screenshots and when one visual QA pass is enough. A new capture guard refuses repeated main baselines and oversized capture sweeps, and every ETA is computed from the remaining queue. See
plugins/vibe/skills/vibe/references/quality-loop-visual-qa.md:83andplugins/vibe/skills/vibe/scripts/capture-guard.mjs:71.AC-1, when to capture:
references/quality-loop-visual-qa.mdgains the blockwhen-to-capturewith the approved criterion 1 text. By default, one visual-qa reviewer reads the diff plus the stories. Captures happen only for a shared component's pixels, for a criterion or ruling that asks for visual proof, or for a finding the reviewer cannot settle.quality-loop.mdand the fleet QA gate point at that block. The verify worker captures only in those cases, or when the session has a reference image for the screen. After fixes, handoff reruns visual QA as one pass scoped to the changes since the last reviewed capture.AC-2, one main baseline: the fleet QA gate and the verify worker compare against the ticket's one main baseline, never a new one because main moved, and a difference outside the ticket's diff is main's. The guard refuses any second main baseline for the ticket.
AC-3 and AC-4, the capture guard. New
skills/vibe/scripts/capture-guard.mjs, with subcommandsbegin,end,statusandfacts.beginrefuses (exit 79):It only refuses; nothing requires a capture.
endrecords ok or failed once, with the measured seconds. A running entry holds its place while its owner lives, or for its own estimate (never under 20 minutes). The ledger is the fleet program'scapturesdirectory when this clone is a writer worktree on exactly one program's board, otherwise the clone's own ledger. The fleet ledger and the clone's ledger mirror each other.Overrides, fixed shape: a refusal is lifted only by a Ruling line in the program rulings, written as
- Ruling YYYY-MM-DD (<source>): "<the operator's exact words>" <TICKET> capture-guard override: <kind>.baselinelifts only the two baseline refusals;budgetlifts only the shot and minute refusals.references/state.mdtells recorders this shape.Fleet slot:
fleet.sh ... slot heavy <owner> <dir> --capture <spec.json> <command...>.rulings.mdbefore the load check and before any slot is taken.--captureis unchanged.AC-5, ETAs: the fleet reporting rule and the Remaining work report template require every ETA for queued work to be computed from the remaining queue count, and restated the moment a step adds more than 30 minutes. See
plugins/vibe/skills/fleet-orchestrator/SKILL.md:212andplugins/vibe/skills/fleet-orchestrator/references/templates.md:126.AC-6: each guard rule has a test that fails on base 8a858bb or under its mutation (see Test plan). See
plugins/vibe/skills/vibe/scripts/capture-guard.test.mjs:80andplugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjs:663.Versions: vibe 1.1.91. vibe-core is unchanged. Instruction words: vibe is about 496 words longer than in 1.1.86 (origin/main 8a858bb), as the CHANGELOG states.
Affected plugins and files
skills/vibe/scripts/capture-guard.mjsand its test.skills/fleet-orchestrator/scripts/fleet.shandfleet.test.mjs.skills/fleet-orchestrator/SKILL.md,references/fleet-qa-gate.md,references/state.mdandreferences/templates.md.skills/vibe/references/quality-loop.mdandquality-loop-visual-qa.md.skills/handoff/SKILL.mdandreferences/handoff-ui.md.agents/vibe-verify-worker.md.tools/python/test_vibe_iss_14158_capture_contract.py.test_pr_takeover_skill_contract.pyandtest_vibe_ws8_load_contract.py.plugin.jsonfiles and the README version line.tools/vibe-bench/src/reply-checker.test.ts: version pin.CHANGELOG.md: vibe 1.1.91.Rulings applied
capture-guard override: baselineorbudgetmarker, and each kind lifts only its own refusals.endafter a capture, and the verify worker's capture rule including the reference image case;--rulingslimit below.Accepted limits
begin --factstrusts the facts it is given. Remote fleet captures read them on the host that holds the worktree.--rulings. The guard cannot verify that a local file is the program rulings or the ticket body. fleet.sh always passes the program rulings.status --ledger <orchDir>/capturesdoes not list direct captures, although the guard refuses across both ledgers for local fleet captures.Overlap
vibe-closedloop-providercopies of the fleet and vibe references need a re-sync from whichever of the two merges second.CHANGELOG.md, both vibeplugin.jsonfiles,plugins/vibe/README.md, and the release pins intest_pr_takeover_skill_contract.py,test_vibe_ws8_load_contract.pyandtools/vibe-bench/src/reply-checker.test.ts.plugins/vibe/skills/fleet-orchestrator/SKILL.md.plugins/vibe/skills/handoff/SKILL.md(a different passage).CHANGELOG.md, both vibeplugin.jsonfiles andplugins/vibe/README.md.CHANGELOG.mdonly.Follow-up outside this repository
pnpm vibe shotsand the proof scripts that call it should run the capture guard'sbeginbefore capturing andendafter, so the guard refuses at the capture command itself.Known main failures
CI job "Vibe Eval T0" (
npm testintools/vibe-bench):tools/vibe-bench/src/phase3.test.tsfails on an expired quarantine deadline. It fails the same way on main 8a858bb; this diff does not touch that file. The recorded lane identity of that job:{"job":{"name":"Vibe Eval T0","runs-on":"ubuntu-latest","steps":[{"uses":"actions/checkout@v4"},{"uses":"actions/setup-node@v4","with":{"node-version":"24"}},{"uses":"astral-sh/setup-uv@v6","with":{"python-version":"3.13"}},{"run":"uv sync --frozen --group dev"},{"name":"Install deterministic benchmark tooling","run":"npm ci","working-directory":"tools/vibe-bench"},{"name":"Type-check benchmark contracts","run":"npm run typecheck","working-directory":"tools/vibe-bench"},{"name":"Validate local plugin structure","run":"npm run validate:plugins","working-directory":"tools/vibe-bench"},{"name":"Run deterministic benchmark contracts","run":"npm test","working-directory":"tools/vibe-bench"},{"name":"Run deterministic Codex mirror contracts","run":"uv run pytest plugins/vibe/evals/codex/test_run.py"}],"timeout-minutes":10},"matrix":{},"version":1,"workflow":{"concurrency":{"cancel-in-progress":true,"group":"ci-${{ github.event.pull_request.number || github.ref }}"},"name":"CI","on":{"pull_request":null,"push":{"branches":["main"]}}}}Test plan
Final candidate tree: tested-head treeSha256
4d21d5084cd477d7f7ac87483cc2ecea09d7d2369d5a7767a4087fe603d1f66b.node --test plugins/vibe/skills/vibe/scripts/capture-guard.test.mjs plugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjspassed 64/64.node --test plugins/vibe/skills/fleet-orchestrator/scripts/generic.test.mjs plugins/vibe/skills/fleet-orchestrator/scripts/slot-lock.test.mjs plugins/vibe/skills/vibe/scripts/skill-split.test.mjspassed.uv run pytest -qon these contracts passed (69 passed):plugins/vibe/tools/python/test_vibe_iss_14158_capture_contract.py,test_vibe_prompt_split_contract.py,test_vibe_1074_lessons_contract.py,test_vibe_1078_lessons_contract.py,test_vibe_1079_lessons_contract.py,test_vibe_110_single_use_version_check_contract.py,test_vibe_connection_restore_contract.py,test_vibe_stranded_writer_recovery_contract.pyandtest_vibe_no_person_names_contract.py.fleet.test.mjsgains two cases, a missing or unreadable--capturespec is a usage error before any load check, slot or command, and--capturewith--containerized-e2e-buildis accepted in either order.node --test plugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjspassed 44/44 on a heavy slot; moving the spec checks after the load check fails the first case, and accepting the flags in one order only fails the second.endrefusing finished entries;\nonly fails the line-split test, and restoring it passes.node plugins/vibe/skills/fleet-orchestrator/scripts/instruction-words.mjs --base 8a858bb4 --changelog CHANGELOG.md --version 1.1.91: net +496, check passes.uv run ruff check .), typecheck (uv run pyright) and pytest (uv run pytest plugins/) all passed.tools/vibe-bench/src/phase3.test.ts, test "local driver snapshots one committed tree and writes one immutable dual-runtime result" (an expired quarantine deadline). The failure is identical on main 8a858bb (see Known main failures).plugins/vibe-core/lib/vibe/scripts/e2e-lock.test.mjs, tests "one SIGINT allows the repository teardown budget to finish without forced escalation" and "a second signal explicitly escalates a stuck command group", in the second machine's run;plugins/vibe/skills/codex-fleet-monitor/scripts/recover.test.mjs, test "a second supervisor on the host exits while the first holds the lock", cancelled at 20 s in a local rerun, and passed in 2.3 s on the second machine.Vibe gate override: ISS-14184: node-tests failed only on timing tests outside this diff under load (the e2e-lock.test.mjs signal tests and recover.test.mjs's second-supervisor test), each passing on another full run or alone; this diff's files passed in every run